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Abstract 


Explicit  expressions  are  derived  for  derivatives  at  zero  lay  of  the  mean 


square  prediction  error  of  a  class  of  random  processes  which  includes  the 
rational  case.  A  simple  random  process  is  demonstrated  which  is  not  in  the 


class.  A  sufficient  condition  is  given  to  determine  whether  a  random  process 


is  in  the  class. 
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Introduction 

Finding  the  minimum  variance  linear  predictor  of  a  continuous  time  wide 
sense  stationary  random  process  is  a. classical  problem  in  the  electrical 
engineering  and  mathematics  literature.  Wiener  [1J  solved  the  problem  of 
prediction  given  that  the  infinite  past  of  the  process  is  observed.  Krein  C2J 
solved  the  problem  in  principle  for  the  much  harder  case  of  observing  only  a 
finite  portion  of  the  entire  past.  The  problem  is  that  the  prediction  formula 
is  still  extremely  difficult  to  calculate  in  a  practical  sense.  Rozanov  L3] 
gives  formulae  for  the  rational  power  spectrum  case  but  these  still  require 
solving  complex  differential  equations  to  obtain  certain  constants.  Given 
that  the  predictors  may  be  difficult  to  calculate  one  may  desire  to  have 
recourse  to  suboptimal  ad-hoc  schemes.  If  so,  it  is  of  crucial  importance  to 
be  able  to  calculate  the  mean  squared  error  performance  of  such  a  suboptimal 
predictor  and  compare  it  to  the  performance  of  the  optimal  one. 

More  recently  Cuzick  [4]  gives  a  nice  overview  of  the  problem  and  derives 
an  upper  bound  to  the  prediction  error  which  approaches  zero  at  the  correct 
rate  as  the  "lag"  or  time  interval  into  the  future  approaches  zero.  These 
bounds  however  have  a  unknown  constant  appearing  in  them  which  hampers  (as  the 
author  states)  the  practical  utility  of  the  results. 

In  this  note  we  will  give  approximate  expressions  for  the  mean  squared 
prediction  error  for  small  extrapolations  into  the  future  for  a  class  of 
random  processes  which  includes  the  rational  case.  Much  of  our  intuition  is 
based  upon  how  filtering  and  prediction  works  for  the  rational  case.  As  an 
interesting  sidelight  it  is  demonstrated  that  the  predictors  (and  their  error) 
can  depart  very  widely  from  those  that  would  be  expected  based  upon  a  rational 
approximation. 


s\ 


Notation 

(1)  {x(t )  -  •  <. -T  £t  £0}  is  an  observation  of  a  wide  sense  stationary 
nondetermini  Stic  zero  mean  random  process. 

(2)  x(k)(t)  is  the  kth  mean  square  derivative  of  x(t)  (if  it  exists). 

(3)  Xj(t)  be  the  minimum  mean  squared  error  predictor  of  x(t),  t  0. 

(4)  E{(xlx)  -  xt(t))2}  *  mseT(x). 

(5)  F * (A)  is  the  power  spectrum  of  the  random  process  x(t).  F  \f* (a )>=R(t) 
the  autocorrelation  function  the  random  process  where  F(  )  and  F  *{  ) 
denote  the  Fourier  and  inverse  Fourier  Transform  operators  respectively. 

(6)  R(t)  ■  f(t)  *  f(-t)  where  f(t)  is  a  causal  square  integrable  function 
which  is  guaranteed  to  exist  by  the  Paley-Wiener  Theorem  [5]. 


Devel opment 

Suppose  F'(x)  is  rational  and  F'(x)  *  U,  then  F{f(t)>  is  also  rational. 


F '  (X )  = 


l  AUiX  )J 
j*0  J 

l  BU1X)  j 
j*0  J 


where  A'  and  B'  are  real,  and  B  >  a+1. 
J  J 


F( f ( t ) ) 


I  AWiX)J 
j»0  J 

I  BUiX)j 
j«0  J 


Let  k  be  such  that  B  >  a+k+1. 


*'  »*,  •* 
vv;.- 


Then  the  Laplace  Transform  of  the  kth  derivative  of  f(t)  is  given  by, 

<Sk)  l 

!(.<*>< t),  ■  ,  -  ■ 

L 

Y=0 

By  the  Abelian  Theorem  for  Laplace  Transforms,  we  have 


f(k)(U)  =  lim  |.{f(k)(t)}s 


0  >  o+k+1 


5?-  (2n)k+1 
B0 


0  =  a+k+1 


However  from  [6,  py.  544]  we  have 


R(k)(U+)  = 


20  >  2a+k+2 


1  a;2 

2 


(2wi) 


20 -2a 


20  «  2a+k+l 


Hence 


f^(O)2  -  2R^Zk+1^(U+)(-l)k+1 


(for  0-a-l  >_  k) 


A  random  process  of  the  above  type  has  exactly  0-a-l  derivatives 

s 

Therefore  we  can  consider  a  suboptimal  predictor  x(r),  of  x(t)  as 


T 


(x)  * 


B-a-1 

I 

k-0 


k 

T 

k! 


x(k>(L.  . 


We  define 


mse(t)  ^  £{(x(t)  -  x  (t))2} 


It  is  straightforward  (but  extremely  tedious)  to  verify  that 


a^mse  (t) 


3t 


k-1 


k+1 


T=0 


+  s  2(|(_1}  R(k)(0+)(-l)  2  k  <  2(8-o)  -1 


A  well  known  formula  [5,6]  for  the  optimum  prediction  when  T  =  »  is  yiven 


by 


rase  (t)  *  E{(x(i)  -  x  (t))2}  *  /  f2(t)dt 

U 


Again  it  is  straightforward  but  tedious  to  verify  that 


3Kmse  (t) 


3t 


t*0 


(*_))(/  2  )(°+))2  k  *  2(8-o)-l 

~T 

3^k  to(i)| 


(we  emphasize  again  that  the  above  Is  zero  unless  k  *  2(8-a)-I).  Hence 
since  mse(t)  represents  the  error  of  a  suboptimal  estimator  which  depends  only 
upon  the  value  of  the  random  process  and  its  derivatives  at  zero  we  must  have 


where  k  <  2(8-o)  -1.  Taking  the  limit  as  t  ♦  0  we  have  our  theorem: 


Theorem  1 

Suppose  the  random  process  x(t)  has  a  rational  power  spectrum  F * (x )  such 
that  F ‘ ( A  )  *  0. 


3  msej(t) 


3t 


k-1 
-  2<k-l> 


k+1 

R(lc)(0+)(-l)_2~ 


t=0 


k  <  2(B-a)-l 


where  R(t)  =  F  _1{F ‘  ( A )} . 


Remark  1 

The  above  development  proves  the  existence  of  the  partial  derivative  and 
gives  its  value  at  t  =  U+.  Note  that  for  all  values  of  K  smaller  than 
2( B-a) -1  the  value  is  zero. 


Remark  2 

By  the  nature  of  the  proof  we  see  that  for  small  prediction  "lays"  the 
simple  Taylor  series  predictor  performs  just  about  as  well  as  the  more  complex 
Wiener  predictor  for  processes  with  rational  power  spectra. 

One  may  now  hope  that  a  more  general  class  of  random  processes  will 
behave  in  the  same  manner.  Since  the  rationals  are  dense  in  the  class  of  all 
spectra  we  could  hope  that  all  random  processes  would  behave  like  this.  This 
is  not  the  case  as  shown  by  the  following  counterexample: 


Counterexample 

Suppose  F ' { A) 


have 


sin2(|)  T 

- 2 - and  j  -  N  a  positive  integer. 


Then  from  [bj  we 


mseT(x)  *  5T2M+1)  T  +  $fciiT  0  <  t  <  1/2 


amseT(r) 


3t 


_  2N+2  .  N+l 

T=0+ "  i2N+i)8  jzmjs 


N+l 

(2N+1 )? 


Suppose  we  use  x(r)  =  x(0)  (the  best  Taylor  Series  predict  or).  Then 


since 


K(X)  =  i  -  {  X 


0 


for  0  j<  x  <  1 
\  >  1 


/I  .  1  n2 
mse(x)  •  j  -  --  * 


1/4 


amse(t) 

3t 

Remark  1 

Obviously  these  two  expressions  are  unequal  with  the  optimal  predictor 
error  derivative  strictly  less  than  1/2  for  all  integer  values  of  N  and 
approaching  1/4  as  N  approaches  «». 


Remark  i 


In  contrast  to  the  rational  case  the  partial  of  the  error  depends  upon 
the  length  of  the  observation  T  we  are  given. 


However  if  we  impose  some  technical  conditions  we  can  demonstrate  an 
entire  class  of  power  spectra  that  will  act  as  in  the  rational  case. 

Theorem  2 

Suppose  f(t)  has  the  following  representation 

t 

f(t)  *  f(0)  +  /  f ' (a)da 
0 


where  r‘(a)  is  square  integrable  i.e.  f (t)  is  absolutely  continuous  and  its 

2 

derivative  is  a  square  integrable  function.  Then  R'(Q+)  •  . 


Proof 

For  t  >  0  we  have 


H( t )  s  /  f(a)f(a-x)da 

T 


=  /  f(a)Lf(a)  -  /  f(a-s)dsjda 
T  0 


(by  absolute  continuity  of  f) 

=  /  f^(a)da  -  /  /  f(o)f 1 (a-s)dods 
r  0  0 

+  /  /  f (a)f 1 (a-s)dads 


(integration  interchange  justified  by  invoking  Tonelli's  Theorem 
on  |f(o)f ' (o-s) |  to  get  integrability  on  cross  measure  then  Fubini's  Theorem 
allows  interchange).  Now  consider  the  last  term  divided  by  t  i.d. 


■7  /  /  f(o)f '(o-s)dods 
T  0  0 


£7  f\  /T  f2(a)da)1/2(/Y2(a-s)da)1/2ds 

‘an  a 


0  0 


(by  Schwartz  inequality) 


£7  /T  K(  /T  f2(o)da)1/2ds 
T  0  0 


-  t  K(mse  (t))1'2  *  mse1/2  K  - ->  0 

T+0 


Therefore  we  may  now  safely  invoke  the  fundamental  theorem  of  the  calculus  and 
find  (for  t  >  0) 


R'(t)  =  -r(x)  -  /  f(#)f'(o-T)do  +  0(x ) 
0 


where  we  have  terms  that  are  approaching  zero  as  t  approaches  zero  by  the 
above  agrument.  Utilizing  the  fact  that  f*  is  square  integrable  we  may 
approximate  it  uniformly  close  by  a  continuous  function  which  implies  we  can 
take  limit  as  t  goes  to  zero  through  positive  values  to  obtain 

R'(0+)  *  -f2(0)  -  /  f(o)f 1 (a)do 
0 

.  .f2,0) 

0 

.  .f2,0)  .1^21.  -1^21 


.*•  V  W  •*  V  v  v,v  ,  .  \ 


vv-%/v‘v’vy-  v  v  *-\v; 


,v.v:  v  v.v  V 


.  •  _  »  _  I 


•  .v  _  •  _ » 


.A 


Note  the  next  to  last  line  follows  since 


00  T 

/  | f ( x ) |  | f ' (x) |dx  =  lim  /  j f ( x ) |  |f'(x)|dx 
0  T+«  0 

<_  ifH  lf'»  *  M  <  » 

which  implies  the  following  limit  exists 

T 

lim  /  f ( x ) f ' (x)dx  =  M‘  <  « 

T+co  o 

? 

because  absolute  summability  implies  summability.  Therefore  lim  f(T) 

Tx» 

exists  and  hence  must  equal  zero. 

Q.E.D. 

Remark  1 

The  theorem  statement  corresponds  to  Eq.  (1)  with  k=0.  We  could  obtain  a 

(k-ll 

statement  for  arbitrary  k  by  making  the  same  assumptions  on  f'  '(t)  that  we 
do  for  f(t).  Although  the  proof  is  messier,  it  is  essentially  the  same. 

Remark  2 

The  counterexample  fails  the  theorem  requirements  since 

f(t)  0  <  t  <  1 

we  would  have  to  have  f‘(t)  =  -  |-  5(t-l)  and  this  candidate  for  the  derivative 
of  course  is  not  square  integrable. 


Remark  3 


One  can  now  generate  entire  families  of  power  spectra  that  behave  as  the 
rational  spectra  do.  For  example  take 


f(t)  =  1 

=  0 

f'(t)  ■  0 


0  <  t  _<  1 

t  >  1 

t  <  0 

0  <  t  <  1 


=  -e 


-(t-1) 


t  >  1 


=  0  t  <  0 

and  It  Is  square  integrable. 

Rx(t)  *  2-t  -  0  <  t  <  1 


3t 


t=0 


f(0)2 

2 


Remark  4 

2  2 

If  feL  and  f'cL  a  well  known  property  of  convolutions  states  that  f*f' 
is  not  only  continuous  but  uniformly  continuous.  Hence  this  implies  the 
derivative  of  the  autocorrelation  function  must  exist  at  every  point  (see 
proof  of  theorem  2)  and  be  continuous  (except  possibly  at  the  origin).  This 

condition  would  usually  be  easier  to  check  on  a  particular  autocorrelation 

o 

function  than  if  f,  f'  e  t  .  Our  counterexample  with  the  triangular  auto¬ 
correlation  of  course  fails  this  test. 


Basically  this  note  reports  on  an  interesting  phenomenon  dealing  with  the 
prediction  error  for  small  lags.  For  the  rational  power  spectrum  case  a 
closed  form  expression  is  derived  for  the  derivative  with  respect  to  the  lag 
of  the  mean  square  prediction  error.  It  is  shown  that,  although  it  appears  a 
large  class  of  "nice”  power  spectra  do  behave  like  the  rationals,  there  are 
also  simple  ones  that  don't,  in  particular  the  triangular  autocorrelation 
function.  For  these  "nice"  random  processes  simple  Taylor  Series  type 
predictors  perform  optimally  well  for  small  lags  into  the  future. 

The  problem  of  whether  a  particular  power  spectrum  is  “nice"  or  not 
remains  an  open  problem.  A  sufficient  condition  given  in  terms  of  smoothness 
attributes  on  the  functions  in  the  Wiener  spectral  decomposition  has  been 
given  but  its  utility  is  limited  due  to  the  difficulty  of  performing  the 
decomposition  and  verifying  the  conditions.  We  are  able  to  remark  though  that 
if  the  autocorrelation  function  isn't  everywhere  differentiable  (except 
possibly  at  zero)  then  it  will  not  be  able  to  meet  our  smoothness  conditions. 
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